[Twisted-Python] Simultaneous Blocking Operations and Twisted
Hi All, I am new to twisted and python. I am evaluating the possibility of using it for an application that I am writing. The framework looks very comprehensive (I am also enjoying the python language :) but I have some questions regarding deferred objects and the twisted web server itself. My application essentially comprises the following: 1. HTTP request is received by the server. 2. Server creates a listening socket on an arbitrary port and waits for n seconds for incoming data 3. Data (or timeout message) is returned to the client by way of the HTTP response. This is a relatively simple application and I have created a prototype in PHP with fastcgi. I am well aware of the limitations of PHP when it some to asynchronous events. The application needs to service thousands of clients simultaneously. In terms of CPU power, this is not a problem, the logic is simple and there is little overhead having thousands of listening sockets waiting for data (I'd prefer to use epoll but select or poll will do). However, there is a massive memory overhead because it is necessary to ensure that there is always one, idle process for any incoming request. Given that the request may wait for up to n seconds before returning a response, the number of simultaneous PHP fastcgi processes that are required is unreasonably high. I have toyed with the idea of writing a small (low memory) binary that will accommodate my needs, but my C isn't great, hence twisted. I was wondering whether it is feasible to use the twisted framework for this application. I have been playing with the framework and I can't see a way to service what are essentially thousands of requests that perform blocking operations simultaneously. I have noted the twisted documentation on threads but I was hoping, for the sake of my sanity, that I might be able to manage this using an asynchronous approach. Essentially, what I think I am looking for is a waitFor() on a deferred callback. As an aside, is it possible to compile platform specific python applications as small binaries? Or does the interpreter (is it required?) mean the memory consumption is prohibitive? Can anybody offer any advice on the matter? Thanks a lot :) Matthew Glubb
On Fri, 30 Mar 2007 12:01:27 +0100, Matthew Glubb <matt@zgroupplc.com> wrote:
Hi All,
I am new to twisted and python. I am evaluating the possibility of using it for an application that I am writing. The framework looks very comprehensive (I am also enjoying the python language :) but I have some questions regarding deferred objects and the twisted web server itself.
My application essentially comprises the following:
1. HTTP request is received by the server. 2. Server creates a listening socket on an arbitrary port and waits for n seconds for incoming data 3. Data (or timeout message) is returned to the client by way of the HTTP response.
[snip]
I was wondering whether it is feasible to use the twisted framework for this application. I have been playing with the framework and I can't see a way to service what are essentially thousands of requests that perform blocking operations simultaneously. I have noted the twisted documentation on threads but I was hoping, for the sake of my sanity, that I might be able to manage this using an asynchronous approach. Essentially, what I think I am looking for is a waitFor() on a deferred callback.
There's no way to block until an event occurs. Blocking would prevent the reactor from noticing the event or informing you about it. :) Fortunately you don't need to resort to threads, either. The conventional way to do this would be to write a function which sets up the listening port and returns a Deferred which will eventually be called back with the result (presumably the data which some client will eventually send to it). Then, using reactor.callLater, set up a timed call which will tear down the listening port after the timeout has expired and errback the Deferred to indicate no data arrived within the timeout period. Actually, that's the only way to do it. :P The conventional implementation would be to do this using explicit callback functions in the style you may have noticed in many Twisted examples. There are other ways which allow you to write code in a different style which some people prefer, but it's just a difference in spelling, the same thing is happening underneath.
As an aside, is it possible to compile platform specific python applications as small binaries? Or does the interpreter (is it required?) mean the memory consumption is prohibitive?
Generally the only thing that's possible is to wrap up the whole application with the whole interpreter and some subset of the standard library in an executable. This doesn't sound much like what you're after, though. Jean-Paul
Hi Jean-Paul, Thanks for your response. On 30 Mar 2007, at 12:18, Jean-Paul Calderone wrote:
The conventional way to do this would be to write a function which sets up the listening port and returns a Deferred which will eventually be called back with the result (presumably the data which some client will eventually send to it). Then, using reactor.callLater, set up a timed call which will tear down the listening port after the timeout has expired and errback the Deferred to indicate no data arrived within the timeout period.
Yes, that's the way I have started doing it, basing it on what I have seen in various modules and sub-classing ClientFactory and LineReceiver. I am a little confused as to how, when the Deferred is returned from the factory, the callbacks that are defined are connected to the originating client request. I assume that all I have to do is to write the response to the client from the callback and that's it. I guess what I am asking is, if an HTTP request (serviced using twisted.web.soap and twistd) starts an asynchronous operation, how does it know that it should wait for an event from the operation before completing its response.
As an aside, is it possible to compile platform specific python applications as small binaries? Or does the interpreter (is it required?) mean the memory consumption is prohibitive?
Generally the only thing that's possible is to wrap up the whole application with the whole interpreter and some subset of the standard library in an executable. This doesn't sound much like what you're after, though.
Yes, that's what I thought, although I wondered whether someone might have written a wonderful tool :) Thanks again for your time, Matt
On Fri, 2007-03-30 at 13:11 +0100, Matthew Glubb wrote:
I guess what I am asking is, if an HTTP request (serviced using twisted.web.soap and twistd) starts an asynchronous operation, how does it know that it should wait for an event from the operation before completing its response.
twisted.web.soap allows you to return a Deferred as a result of a SOAP method. When someone eventually calls .callback() or .errback() on that Deferred the HTTP request will get its response.
Great. That's what I wanted to know, thanks :) On 30 Mar 2007, at 13:37, Itamar Shtull-Trauring wrote:
On Fri, 2007-03-30 at 13:11 +0100, Matthew Glubb wrote:
I guess what I am asking is, if an HTTP request (serviced using twisted.web.soap and twistd) starts an asynchronous operation, how does it know that it should wait for an event from the operation before completing its response.
twisted.web.soap allows you to return a Deferred as a result of a SOAP method. When someone eventually calls .callback() or .errback() on that Deferred the HTTP request will get its response.
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
participants (3)
-
Itamar Shtull-Trauring -
Jean-Paul Calderone -
Matthew Glubb