[Twisted-Python] Problem with trial using the reactor
Hello, Here is a problem regarding the distributed trial test runner. The slave instances receive data from the master, telling them which tests to run. This requires use of the reactor on the slave-side. Unfortunately, when the testCase does the cleanup of the reactor, it cleans up everything including the trial slave. Any ideas on what should be done about this? Thank you, Alex
On 6/7/07, Alex Lang <alex.lang@mail.mcgill.ca> wrote:
Hello,
Here is a problem regarding the distributed trial test runner. The slave instances receive data from the master, telling them which tests to run. This requires use of the reactor on the slave-side. Unfortunately, when the testCase does the cleanup of the reactor, it cleans up everything including the trial slave.
Any ideas on what should be done about this?
This is a roundabout way of answering your question, but it's helpful for me to write it down, and I think it will inform the discussion. In terms of Trial's requirements: * We want to write and run tests that perform asynchronous operations, which essentially means tests that return Deferreds. * We want to be able to run these tests in non-Trial test runners. * We want tests to be isolated as much as possible. * We want to be able to use Trial to test the reactor itself. Some awkward facts: * There is exactly one reactor per Twisted process. This reactor cannot be cleanly restarted in a cross-platform way. * The current Twisted test suite (i.e. what is loaded when you do 'trial twisted') does not play nicely with the reactor. Changing this will take a significant amount of work.[1] So, in response to all of these things: * Trial presents a blocking interface for running any given test: TestCase.run(). This allows Deferred-returning tests to be used in non-Twisted test runners. * By design, Trial totally obliterates the reactor in each call to TestCase.run(). This helps ensure test isolation * Trial's design operates on the assumption that the caller of TestCase.run() will not be doing anything with the reactor. However, all of this presents problems: * The vast bulk of Twisted's code base cannot be used to implement a Trial runner. This is what Alex is discovering now. * Trial's cleanup code is buggy and fragile. Changing anything is likely to break hundreds of tests. * Tests that use twisted.trial.unittest.TestCase run in a highly-specialized, somewhat unrealistic environment. Various solutions have been proposed: * Using a customized reactor wrapper in trial that tracks resources created by tests and cleans up only those resources. * Making reactors restartable. * Allowing for multiple running reactors. i.e. A reactor that could be started within another running reactor. I believe that only the third of these solutions would actually help solve Alex's problem. However, as I understand it, the solution would require extensive API changes across most of Twisted, as things which once assumed a global reactor must now take 'reactor' as a parameter. I'd rather not offer any recommendations in this email. It's difficult enough figuring out what the problem is and what the options are. jml [1] For example, Trial does two calls to 'reactor.iterate()' in its cleanup. If these calls are removed, tests fail and there are a significant number of cleanup errors. I discussed some of the issues here in an email last year.
I just spent some time tearing my hair out getting trial and ThreadedSelectReactor to cooperate. Another possible solution to the trial/reactor impedance mismatch you commented on would be for trial spawn off a process for each test. By definition, that process would get its own reactor and could not taint any other tests. The test rig process could happily use Deferred semantics, while presenting a synchronous interface to trial. This would also be easier to distribute IMO, because if IPC is already involved in communication between trial runner and tests, making that IPC go to another machine is easier. I realize the cost of spawning processes, but I honestly think that it's easier to change unittest to fork, run a test on a separate reactor, and IPC the test results back to the parent, than it is to take one of the other three approaches, because of the API changes they would require. (Not that said API changes would be bad, but I don't think that the distributed runner should have to wait for them.) Ben -- <http://artins.org/ben> "Computers! All they ever think of is hex!"
On 6/7/07, Ben Artin <ben@artins.org> wrote:
I just spent some time tearing my hair out getting trial and ThreadedSelectReactor to cooperate.
Sorry. If it doesn't have a buildslave, I can't support it. And IIRC, we have decided not to support TSR.
Another possible solution to the trial/reactor impedance mismatch you commented on would be for trial spawn off a process for each test. By definition, that process would get its own reactor and could not taint any other tests. The test rig process could happily use Deferred semantics, while presenting a synchronous interface to trial.
Right. I should have included this in my list, as it is both valid and has been suggested before. Thanks, jml
On Thu, 2007-06-07 at 01:23 -0400, Ben Artin wrote:
Another possible solution to the trial/reactor impedance mismatch you commented on would be for trial spawn off a process for each test. By definition, that process would get its own reactor and could not taint any other tests. The test rig process could happily use Deferred semantics, while presenting a synchronous interface to trial.
This is a I-think-so-too post. Letting the OS clean up things is a sane approach. I also see a few more benefits, such as running tests in parallel on SMP machines. Also, I don't think that the overhead of forking/spawning is big enough to pay attention - at worst, 50ms per test and 10 more megabytes of memory. As the saying goes, "We all have pentiums now..."
participants (4)
-
Alex Lang -
Ben Artin -
Jonathan Lange -
Konrads Smelkovs