[Twisted-Python] A Solution Re: [Stackless] Advice on Debugging Stackless/Twisted Problem
Hello Colleagues: Earlier I posted that I was having a problem involving the loss of deferred calls in a programme that uses Stackless and Twisted. I found the problem and wrote a solution. I think it interesting. It helped when I started using the Twisted logging facility :-) The immediate reason the deferred were "lost" was because the main reactor loop stopped iterating. What happened was: 1. In the example, Stackless treats Twisted as a tasklet. see: http://www.stackless.com/pipermail/stackless/2007-April/002448.html 2. When the Request Handler is created, it is running as a *part* of the Twisted tasklet. 3. Consequently when the Request Handler blocked on a channel, this blocked the entire reactor! In turn, doIteration() (doSelect) cannot do its work. Twisted grinds to a halt. 4. When the last tasklet (TestTasklet) blocked, all the tasklets became blocked. Consequently the programme ended. The solution was two-fold. 1) To prevent the reactor from blocking, I run the request handler in its own tasklet. This prevents the reactor from blocking for indeterminate periods of time. 2) I modified doSelect to call stackless.schedule(). If the reactor does not call stackless.schedule(), other tasklets will not get an opportunity to run. (When I have time, I will write a proper reactor. This should be a little more than overriding the doIteration() method of selectReactor). Or re-read the Blocking demo : I don't quite understand how it works) With this approach, I 1) Eliminate a OS-Thread (both Stackless and Twisted folks concur that this is wrong, wrong, wrong). 2) Can execute tasklets in parallel with tasklets that are blocked waiting for network I/O like an incoming HTTP request. I think it is neat that the Twisted framework is flexible enough to be contorted in this fashion. Unfortunately this approach is *very* slow. I feel it is wastefully that a tasklet is created for every incoming request. I have to work on this. ~ I understand the subset of Stackless and Twisted users is small. And the philosophies are different. And what I am doing is a bit esoteric and application specific. Hopefully others will be able to build upon this : I don't see why programmers can't get the best of the Stackless and Twisted worlds. Cheers, Andrew ____________________________________________________________________________________Yahoo! oneSearch: Finally, mobile search that gives answers, not web links. http://mobile.yahoo.com/mobileweb/onesearch?refer=1ONXIC
2) I modified doSelect to call stackless.schedule(). If the reactor does not call stackless.schedule(), other tasklets will not get an opportunity to run.
Why not just use twisted.internet.task.LoopingCall?
On 05:39 pm, andrewfr_ice@yahoo.com wrote:
(When I have time, I will write a proper reactor. This should be a little more than overriding the doIteration() method of selectReactor). Or re-read the Blocking demo : I don't quite understand how it works)
I've said it before and I'll say it again. Writing a reactor is the _wrong_ way to attempt to integrate Twisted and stackless. Reactors are implemented to implement operating system or platform multiplexing technologies, and stackless does not have its own multiplexor. Multiplexing technologies are things like kqueue, epoll, IOCP, WaitForMultipleObjects, and AIO: APIs that allow you to wait for multiple input/output sources (including the clock) at the same time. The issue is confused somewhat by GUI toolkits, which have their own event loops that wrap these APIs, so we have a few things that use those reactors as a back-end to cooperate in the most friendly way possible with those GUIs, but ultimately what Twisted is talking to in GUI libraries is another multiplexing API. Stackless does not have any such API, and nor should it. Making a stackless reactor would needlessly couple stackless/twisted integration to a particular multiplexor. For example, it would make it impossible to write an application which used both GTK and Stackless. Also, you'd need to choose either a portable, but inefficient and non-scalable API (like select) or a highly performant, but highly platform-specific API (like kqueue). This is a decision that should be made when an application is deployed, and only required by an application developer if they are using platform-specific features (which stackless is not). I keep hearing this idea of a "stackless reactor" bandied about, and I don't know how I can make this any more clear. Again I refer to my original post on the subject: http://article.gmane.org/gmane.comp.python.twisted/13946
1) Eliminate a OS-Thread (both Stackless and Twisted folks concur that this is wrong, wrong, wrong).
Notwithstanding the above, I'm glad that you've eliminated this. It's a step in the right direction.
participants (3)
-
Andrew Francis -
glyph@divmod.com -
Itamar Shtull-Trauring