[Python-Dev] Allowing to run certain regression tests in subprocesses

[Eli Bendersky]

On Sun, Aug 4, 2013 at 5:44 AM, R. David Murray <rdmurray at bitdance.com> wrote:
> On Sat, 03 Aug 2013 19:04:21 -0700, Eli Bendersky <eliben at gmail.com> wrote:
>> On Sat, Aug 3, 2013 at 6:57 PM, R. David Murray <rdmurray at bitdance.com> wrote:
>> > On Sat, 03 Aug 2013 16:47:37 -0700, Eli Bendersky <eliben at gmail.com> wrote:
>> >> On Sat, Aug 3, 2013 at 4:36 PM, Eli Bendersky <eliben at gmail.com> wrote:
>> >> > Hi All,
>> >> >
>> >> > Today the issue of cross-test global env dependencies showed its ugly
>> >> > head again for me. I recall a previous discussion
>> >> > (http://mail.python.org/pipermail/python-dev/2013-January/123409.html)
>> >> > but there were many more over the years.
>> >> >
>> >> > The core problem is that some tests modify the global env
>> >> > (particularly importing modules) and this sometimes has adverse
>> >> > effects on other tests, because test.regrtest runs all tests in a
>> >> > single process. In the discussion linked above, the particular culprit
>> >> > test__all__ was judged as a candidate to be moved to a subprocess.
>> >> >
>> >> > I want to propose adding a capability to our test harness to run
>> >> > specific tests in subprocesses. Each test will have some simple way of
>> >> > asking to be run in a subprocess, and regrtest will concur (even when
>> >> > running -j1). test__all__ can go there, and it can help solve other
>> >> > problems.
>> >> >
>> >> > My particular case is trying to write a test for
>> >> > http://bugs.python.org/issue14988 - wherein I have to simulate a
>> >> > situation of non-existent pyexpat. It's not hard to write a test for
>> >> > it, but when run in tandem with other tests (where C extensions loaded
>> >> > pyexpat) it becomes seemingly impossible to set up. This should not be
>> >> > the case - there's nothing wrong with wanting to simulate this case,
>> >> > and there's nothing wrong in Python and the stdlib - it's purely an
>> >> > artifact of the way our regression suite works.
>> >> >
>> >> > Thoughts?
>> >> >
>> >> > Eli
>> >>
>> >> FWIW the problem is also discussed here:
>> >> http://bugs.python.org/issue1674555, w.r.t. test_site
>> >
>> > Can't you just launch a subprocess from the test itself using script_helpers?
>> >
>>
>> I can, but such launching will be necessarily duplicated across all
>> tests that need this functionality (test_site, test___all__, etc).
>> Since regrtest already has functionality for launching whole
>> test-suites in subprocesses, it makes sense to reuse it, no?
>
> In the case of test_site and test___all___ we are talking about running
> the entire test file in a subprocess.  It sounds like you are only
> talking about running one individual test function in a subprocess,
> for which using script_helpers seems the more natural solution.

I was actually planning to split this into a separate test file to
make the process separation more apparent. And regardless, the
question sent to the list is about the generic approach, not my
particular problem. Issues of folks struggling with inter-test
dependencies through global state modification come up very often.

Eli