[Twisted-Python] Control the number of parallel system calls
I've done something very similar. I control the number of concurrent child processes by simply keeping a counter of how many are active at any one time. The main loop that polls a queue will only accept new jobs if the counter is less than the configuration-defined limit. I didn't find anything Twisted-specific to limit the number of forked processes, but it was easy to implement in my own logic. Cheers, Chris On 19 Sep 2007, at 11:26, Daniel de la Cuesta wrote:
Hi,
I am developing a video conversion server. To process the video conversion I use the method "getProcessOutput" to call "ffmpeg".
The user upload the video using a HTTP POST and each video must be converted to 8 or 10 output formats.
Currently, all the conversions for each video are processed in parallel, what I want to do is to control the number of parallel system calls to "ffmpeg". For example I only want 2 processes at the same time and when one of them finishes the next process start.
How can I do that?
Is there any function in Twisted that implements that?
Thank you
In fact the code I use to perform concurrent processing from a disk- based request queue is available here, http://www.psychofx.com/directory_queue_service/ Feedback welcome. Cheers, Chris Miles On 19 Sep 2007, at 11:57, Chris Miles wrote:
I've done something very similar. I control the number of concurrent child processes by simply keeping a counter of how many are active at any one time. The main loop that polls a queue will only accept new jobs if the counter is less than the configuration- defined limit.
I didn't find anything Twisted-specific to limit the number of forked processes, but it was easy to implement in my own logic.
Cheers, Chris
On 19 Sep 2007, at 11:26, Daniel de la Cuesta wrote:
Hi,
I am developing a video conversion server. To process the video conversion I use the method "getProcessOutput" to call "ffmpeg".
The user upload the video using a HTTP POST and each video must be converted to 8 or 10 output formats.
Currently, all the conversions for each video are processed in parallel, what I want to do is to control the number of parallel system calls to "ffmpeg". For example I only want 2 processes at the same time and when one of them finishes the next process start.
How can I do that?
Is there any function in Twisted that implements that?
Thank you
Daniel de la Cuesta wrote:
Hi,
I am developing a video conversion server. To process the video conversion I use the method "getProcessOutput" to call "ffmpeg".
The user upload the video using a HTTP POST and each video must be converted to 8 or 10 output formats.
Currently, all the conversions for each video are processed in parallel, what I want to do is to control the number of parallel system calls to "ffmpeg". For example I only want 2 processes at the same time and when one of them finishes the next process start.
How can I do that?
Is there any function in Twisted that implements that?
The obvious way to do that would be using a twisted.internet.defer.DeferredSemaphore. -Andrew.
Andrew Bennetts wrote:
Daniel de la Cuesta wrote:
Hi,
I am developing a video conversion server. To process the video conversion I use the method "getProcessOutput" to call "ffmpeg".
The user upload the video using a HTTP POST and each video must be converted to 8 or 10 output formats.
Currently, all the conversions for each video are processed in parallel, what I want to do is to control the number of parallel system calls to "ffmpeg". For example I only want 2 processes at the same time and when one of them finishes the next process start.
How can I do that?
Is there any function in Twisted that implements that?
The obvious way to do that would be using a twisted.internet.defer.DeferredSemaphore.
Or http://foss.eepatents.com/AsynQueue ... :-)
Daniel de la Cuesta wrote:
Hi,
I am developing a video conversion server. To process the video conversion I use the method "getProcessOutput" to call "ffmpeg".
The user upload the video using a HTTP POST and each video must be converted to 8 or 10 output formats.
Currently, all the conversions for each video are processed in parallel, what I want to do is to control the number of parallel system calls to "ffmpeg". For example I only want 2 processes at the same time and when one of them finishes the next process start.
How can I do that?
Is there any function in Twisted that implements that?
Thank you
I wrote something similar to what you want some time ago, although I don't think I ever actually used it in the end. See attached module. I included a __main__ section in the module that hopefully demonstrates its use. Note that it was written as a Service so it needs starting, hence the strange delayed call to startService before the reactor is run. Basically, the thing is a service that makes the normal getProcessOutput and getProcessValue utilities available. However, the execution requests are queued and the maximum number of processes allowed to run is limited. Hope it helps. - Matt
Daniel de la Cuesta wrote:
Hi,
I am developing a video conversion server. To process the video conversion I use the method "getProcessOutput" to call "ffmpeg".
The user upload the video using a HTTP POST and each video must be converted to 8 or 10 output formats.
Currently, all the conversions for each video are processed in parallel, what I want to do is to control the number of parallel system calls to "ffmpeg". For example I only want 2 processes at the same time and when one of them finishes the next process start.
How can I do that?
Is there any function in Twisted that implements that?
Hey Daniel, There's a semaphore convenience in twisted.internet.defer - which can be used to control the number of things which have concurrent access to .. something. You could make that something your function which launches a system call, with something like: s = defer.DeferredSemaphore(2) dlist = [ s.run(encodeToFormat, video, format) for format in formats] d = defer.gatherResults(dlist) return d cheers, Andy.
participants (6)
-
Andrew Bennetts -
Andy Gayton -
Chris Miles -
Daniel de la Cuesta -
Ed Suominen -
Matt Goodall