[Twisted-Python] pyprocessing in twisted
Hello list, probably a dumb question, but is it possible / a good idea to use pyProcessing[1] in a Twisted application? More specifically, I'd like to run a Twisted process A that accepts data from the network and sends it down to a process B through a processing.Queue(). Is that wise? Or should I just stick with reactor.spawnProcess() and communicate with ProcessProtocol? Sorry for the newb-ness, peter. [1] http://pyprocessing.berlios.de/
On Fri, 6 Jun 2008 18:21:45 +0200, Peter Sabaini <peter@sabaini.at> wrote:
Hello list,
probably a dumb question, but is it possible / a good idea to use pyProcessing[1] in a Twisted application?
More specifically, I'd like to run a Twisted process A that accepts data from the network and sends it down to a process B through a processing.Queue().
Is that wise? Or should I just stick with reactor.spawnProcess() and communicate with ProcessProtocol?
Why are you interested in doing this? Jean-Paul
On Friday 06 June 2008 18:24:57 Jean-Paul Calderone wrote:
On Fri, 6 Jun 2008 18:21:45 +0200, Peter Sabaini <peter@sabaini.at> wrote:
Hello list,
probably a dumb question, but is it possible / a good idea to use pyProcessing[1] in a Twisted application?
More specifically, I'd like to run a Twisted process A that accepts data from the network and sends it down to a process B through a processing.Queue().
Is that wise? Or should I just stick with reactor.spawnProcess() and communicate with ProcessProtocol?
Why are you interested in doing this?
I'd like to isolate code running in "process B" -- for now this is just a prototype, but later on it could well be code I have not much control over, so I don't want it to interact bad with Twisted. And I want to be able to signal it etc. separately from the Twisted process. PyProcessing seemed like an easy way to do this. peter.
Jean-Paul
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On 04:37 pm, peter@sabaini.at wrote:
I'd like to isolate code running in "process B" -- for now this is just a prototype, but later on it could well be code I have not much control over, so I don't want it to interact bad with Twisted. And I want to be able to signal it etc. separately from the Twisted process. PyProcessing seemed like an easy way to do this.
It's doubtful that it will work. There have been a couple of attempts at fixing this ticket: http://twistedmatrix.com/trac/ticket/733 I've been working on it most recently, and although there will be probably be a fix merged that works with popen sometime soon, I don't know if that will be enough for pyprocessing.
Hi Peter, On Fri, 06 Jun 2008 11:37:41 -0500, Peter Sabaini <peter@sabaini.at> wrote: [snip]
I'd like to isolate code running in "process B" -- for now this is just a prototype, but later on it could well be code I have not much control over, so I don't want it to interact bad with Twisted. And I want to be able to signal it etc. separately from the Twisted process. PyProcessing seemed like an easy way to do this.
peter.
I'd definitely advise sticking with reactor.spawnProcess; using that, along with a ProcessProtocol implementation, is going to allow you to integrate more smoothly with the rest of your Twisted app. The catch here is that, depending upon your needs, you may have tp write your own ProcessProtocol, specific to your app; if you want something that provides more "out-of-box" functionality, I might suggest you look at ampoule, https://launchpad.net/ampoule/, which gives you a process pool, and handles most of the low-level stuff. Hope this helps, L. Daniel Burr
Thanks for the all the suggestions! /me is going into exploring mode peter. On Friday 06 June 2008 18:51:19 L. Daniel Burr wrote:
Hi Peter,
On Fri, 06 Jun 2008 11:37:41 -0500, Peter Sabaini <peter@sabaini.at> wrote:
[snip]
I'd like to isolate code running in "process B" -- for now this is just a prototype, but later on it could well be code I have not much control over, so I don't want it to interact bad with Twisted. And I want to be able to signal it etc. separately from the Twisted process. PyProcessing seemed like an easy way to do this.
peter.
I'd definitely advise sticking with reactor.spawnProcess; using that, along with a ProcessProtocol implementation, is going to allow you to integrate more smoothly with the rest of your Twisted app.
The catch here is that, depending upon your needs, you may have tp write your own ProcessProtocol, specific to your app; if you want something that provides more "out-of-box" functionality, I might suggest you look at ampoule, https://launchpad.net/ampoule/, which gives you a process pool, and handles most of the low-level stuff.
Hope this helps,
L. Daniel Burr
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Take a look at my AsynQueue project (http://foss.eepatents.com/AsynQueue), which includes provisions for this very thing in the asynqueue.workers module. A specific implementation for asynchronous cluster processing is available in my (you guessed it) AsynCluster project (http://foss.eepatents.com/AsynQueue). I'm using it regularly to run a complex statistical model on a 10-CPU cluster here. Best regards, Ed Peter Sabaini wrote:
Hello list,
probably a dumb question, but is it possible / a good idea to use pyProcessing[1] in a Twisted application?
More specifically, I'd like to run a Twisted process A that accepts data from the network and sends it down to a process B through a processing.Queue().
Is that wise? Or should I just stick with reactor.spawnProcess() and communicate with ProcessProtocol?
Sorry for the newb-ness, peter.
[1] http://pyprocessing.berlios.de/
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
participants (5)
-
Ed Suominen -
glyph＠divmod.com -
Jean-Paul Calderone -
L. Daniel Burr -
Peter Sabaini