For a similar use case, we're using a combination of cooperator with defertoprocess. I can explain more if anyone's interested. Le 29 oct. 2011 00:41, "Donal McMullan" <donal.mcmullan@gmail.com> a écrit :
I think twisted.internet.defer.DeferredSemaphore is designed for this. John Paul Calderone answers a similar question in detail:
http://stackoverflow.com/questions/2861858/queue-remote-calls-to-a-python-tw...
Donal McMullan
On 29/10/2011, at 11:21 AM, Daryl Herzmann wrote:
Hi,
I have an application that ingests data and does a reactor.spawnProcess() for each chunk of data (a product) to pass it as STDIN to a compiled binary and then I harvest the STDOUT. It has been working well, but I have an issue when my data rates get too high and the machine gets overloaded with spawned processes (I think) and starts running out of file descriptors (yes I can raise it :) causing all sorts of pain. I'm wondering about a mechanism to throttle the number of spawned processes going at one time? It'd be nice to only have 10 of these spawned processes going at any one time. Thanks for your ideas :)
daryl
code snipet:
class SHEFIT(protocol.ProcessProtocol): def __init__(self, tp): self.tp = tp self.data = ""
def connectionMade(self): self.transport.write( self.tp.raw ) self.transport.closeStdin()
def outReceived(self, data): self.data = self.data + data
def errReceived(self, data): print "errReceived! with %d bytes!" % len(data) print data
def outConnectionLost(self): really_process(self.tp, self.data)
def got_product(): shef = SHEFIT( tp ) reactor.spawnProcess(shef, "shefit", ["shefit"], {})
def really_process(tp,data): print 'Do some work'
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python