On Sun, Aug 4, 2013 at 5:44 AM, R. David Murray firstname.lastname@example.org wrote:
On Sat, 03 Aug 2013 19:04:21 -0700, Eli Bendersky email@example.com wrote:
On Sat, Aug 3, 2013 at 6:57 PM, R. David Murray firstname.lastname@example.org wrote:
On Sat, 03 Aug 2013 16:47:37 -0700, Eli Bendersky email@example.com wrote:
On Sat, Aug 3, 2013 at 4:36 PM, Eli Bendersky firstname.lastname@example.org wrote:
Today the issue of cross-test global env dependencies showed its ugly head again for me. I recall a previous discussion (http://mail.python.org/pipermail/python-dev/2013-January/123409.html) but there were many more over the years.
The core problem is that some tests modify the global env (particularly importing modules) and this sometimes has adverse effects on other tests, because test.regrtest runs all tests in a single process. In the discussion linked above, the particular culprit test__all__ was judged as a candidate to be moved to a subprocess.
I want to propose adding a capability to our test harness to run specific tests in subprocesses. Each test will have some simple way of asking to be run in a subprocess, and regrtest will concur (even when running -j1). test__all__ can go there, and it can help solve other problems.
My particular case is trying to write a test for http://bugs.python.org/issue14988 - wherein I have to simulate a situation of non-existent pyexpat. It's not hard to write a test for it, but when run in tandem with other tests (where C extensions loaded pyexpat) it becomes seemingly impossible to set up. This should not be the case - there's nothing wrong with wanting to simulate this case, and there's nothing wrong in Python and the stdlib - it's purely an artifact of the way our regression suite works.
FWIW the problem is also discussed here: http://bugs.python.org/issue1674555, w.r.t. test_site
Can't you just launch a subprocess from the test itself using script_helpers?
I can, but such launching will be necessarily duplicated across all tests that need this functionality (test_site, test___all__, etc). Since regrtest already has functionality for launching whole test-suites in subprocesses, it makes sense to reuse it, no?
In the case of test_site and test___all___ we are talking about running the entire test file in a subprocess. It sounds like you are only talking about running one individual test function in a subprocess, for which using script_helpers seems the more natural solution.
I was actually planning to split this into a separate test file to make the process separation more apparent. And regardless, the question sent to the list is about the generic approach, not my particular problem. Issues of folks struggling with inter-test dependencies through global state modification come up very often.