Re: [Twisted-Python] Using reactor with multiple processes
Dimitri Vorona wrote:
Hi,
i'd like to be able to start additional instances of my server using multiprocess module. To do it i just start a new Process with mutiprocessing and then create a new instance of my Application there. Everything works fine, but i can't figure out a way to stop such a process. If I run reactor.stop() in the child process I get an infinite loop of errors looking like this:
This is a very timely discussion for us. We are trying to launch multiple independent twistedized children with python multiprocess and yes (as Phil noted) we are getting very strange results. We were aware the multiprocess module does a fork to spawn children but we hoping that this would not be an issue for us because our parent process does NOT start a reactor before spawning the children. In fact even if our parent process never starts its reactor, as soon as we spawn the 2nd twistedized child, bad things start happening in the children. The parent does import the child modules in order to start them and since each child module also imports twisted, we are thinking that some twisted state gets established just on the import of the twisted reactor into the parent and then the fork clones that state into the children. Thanks Phil for the reference to ampoule. We will look at that next. We are still hoping to get the parent set up in such a way that whatever the forking :-) shared state is, it does not get established until after the children are spawed. I'll post back later with what we find. Phil your thoughts on this would also be appreciated. Cheers, David
On Sep 16, 2009, at 12:33 PM, David Yoakley wrote:
Thanks Phil for the reference to ampoule. We will look at that next. We are still hoping to get the parent set up in such a way that whatever the forking :-) shared state is, it does not get established until after the children are spawed. I'll post back later with what we find. Phil your thoughts on this would also be appreciated.
It think it would be a nice for this to work. Importing (but not running) twisted should ideally not cause strange side effects. So if this turns out to be a problem with some twisted-internal code rather than your code, a bug report would be a good idea, and a patch causing such problematic setup to be done at reactor startup rather than import time even better. That should at least fix the case of forking before starting the reactor. For people who want to fork while the reactor is running, there would need to be API for clearing out all the timed events, open sockets, etc and starting over. Which is probably more trouble than it's worth. James
James Y Knight wrote:
On Sep 16, 2009, at 12:33 PM, David Yoakley wrote:
Thanks Phil for the reference to ampoule. We will look at that next. We are still hoping to get the parent set up in such a way that whatever the forking :-) shared state is, it does not get established until after the children are spawed. I'll post back later with what we find. Phil your thoughts on this would also be appreciated.
It think it would be a nice for this to work. Importing (but not running) twisted should ideally not cause strange side effects.
Well, doing: from twisted.internet import reactor ...causes all kinds of code to be executed. I suspect that: posixreactorbase.installWaker() ...is the problem, since this creates an FD pair and adds it to the reactor. I guess this should really be done during reactor.run()
participants (3)
-
David Yoakley
-
James Y Knight
-
Phil Mayers