Concurrency in a twisted.web application

Dear twisted.web developpers and users, I have written a twisted application for viewing and exporting movies. These movies are in a format specific to my company. I use : twisted 2.1.0, twisted.web 0.5.0, and nevow 0.5.0 (all html templates are written using nevow). In this web application, there is a feature for exporting a movie file into mpeg, and sending it to the client web browser (file download). This feature is handled by a resource object which is a subclass of static.File. Exporting a movie into mpeg implies : -> opening the original video file -> doing the conversion (which can take as long as 1 minute or more) -> sending the resulting mpeg file to the browser (file download) The problem is that while the twisted web application is working handling the export request, the twisted web server is not responsive to any other HTTP request, as long as the video conversion and file transfer is not finished. I would like to know if there is a standard or recommanded way of setting up concurrency into a twisted.web application. And also if there is a sample program available somewhere. I would like my web application to be still responsive to other HTTP requests while : -> file conversion to mpeg is being processing -> mpeg file is being transfered to the client browser (which is handled by a static.File subclass resource) Thank you very much for all your answers. I appreciate very much working with twisted.web and interested to get more insight into the way twisted core handles requests. Anand

Ananda Tallur wrote:
The problem is that while the twisted web application is working handling the export request, the twisted web server is not responsive to any other HTTP request, as long as the video conversion and file transfer is not finished.
If you are running Linux, you _really_ want to isolate the movie conversion to another process. reactor.spawnProcess. (Yes, process, not thread.)

I have sent a question a few days ago, concerning having concurrency in a twisted.web application. I have a web application, which has some features which imply doing long calculations. My need would be to have the web application still responsive to other http requests, while processing a calculation triggered by another request. Therefore, more preciserly, for a request which needs to trigger calculations, I would like the following scenario to take place : -> the resource serving the request launches a dedicated thread or process to perform the calculation -> the resource returns a defered, so that twisted.web core gets the control back and continues serving other requests -> when the calculation thread/process has finished its calculation, a callback is called to finish processing the request. I have no idea of how to implement this, and can't find a documentation or an example for this kind of concurrent web application with twisted.web. Does anyone has already written something close, or knows a documentation / example which could help me ? I really thank everyone in advance. Anand On 16 nov. 05, at 11:50, Ananda Tallur wrote:
Dear twisted.web developpers and users,
I have written a twisted application for viewing and exporting movies. These movies are in a format specific to my company. I use : twisted 2.1.0, twisted.web 0.5.0, and nevow 0.5.0 (all html templates are written using nevow).
In this web application, there is a feature for exporting a movie file into mpeg, and sending it to the client web browser (file download).
This feature is handled by a resource object which is a subclass of static.File. Exporting a movie into mpeg implies : -> opening the original video file -> doing the conversion (which can take as long as 1 minute or more) -> sending the resulting mpeg file to the browser (file download)
The problem is that while the twisted web application is working handling the export request, the twisted web server is not responsive to any other HTTP request, as long as the video conversion and file transfer is not finished.
I would like to know if there is a standard or recommanded way of setting up concurrency into a twisted.web application. And also if there is a sample program available somewhere.
I would like my web application to be still responsive to other HTTP requests while : -> file conversion to mpeg is being processing -> mpeg file is being transfered to the client browser (which is handled by a static.File subclass resource)
Thank you very much for all your answers. I appreciate very much working with twisted.web and interested to get more insight into the way twisted core handles requests.
Anand
_______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web

On Tue, Nov 29, 2005 at 02:13:20PM +0100, Ananda Tallur wrote: [...]
I have no idea of how to implement this, and can't find a documentation or an example for this kind of concurrent web application with twisted.web.
Tommi's advice to use reactor.spawnProcess is good. You may find the "mirror" component of Backwards to be helpful -- if I recall correctly, if a thumbnail is requested but doesn't exist yet, it can spawn a process to generate it, which sounds very similar to what you are after. Backwards can be found at: http://puzzling.org/computing/software/projects/backwards -Andrew.
participants (3)
-
Ananda Tallur
-
Andrew Bennetts
-
Tommi Virtanen