[Twisted-Python] Better usage of Twisted on multi core processor
Hi all, I have a twisted TCP server to listens to client, processes requests, do mysql database operations if needed (using adbapi Connection pool) and return the result. Before deploying this in production, I want to know right way to configure the server. Since twisted is single threaded, how can I leverage multiple cores of my production machine (which has 6 cores and 16 GB RAM) ? One approach that I thought of was to start multiple instances of twisted server on different ports. This would help in using the other cores as well. What do you guys suggest ? Thanks -Sumanth
Hi Sumanth, The first answer by JP Calderone (exarkun) on the following stackoverflow question may be of some assistance. http://stackoverflow.com/questions/10077745/twistedweb-on-multicore-multipro... This answer is from 2012; the good news is that the good API that JP talks about in his answer is now available in released Twisteds IIUC :) cheers lvh
Nice. Thanks for the info. On Jan 20, 2014 8:02 PM, "Laurens Van Houtven" <_@lvh.io> wrote:
Hi Sumanth,
The first answer by JP Calderone (exarkun) on the following stackoverflow question may be of some assistance.
http://stackoverflow.com/questions/10077745/twistedweb-on-multicore-multipro...
This answer is from 2012; the good news is that the good API that JP talks about in his answer is now available in released Twisteds IIUC :)
cheers lvh
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Hi, On 01/20/14 15:30, Laurens Van Houtven wrote:
The first answer by JP Calderone (exarkun) on the following stackoverflow question may be of some assistance.
http://stackoverflow.com/questions/10077745/twistedweb-on-multicore-multipro... What is the best way to use this capability from a .tac file?
On Mar 11, 2015, at 11:57 PM, Nagy, Attila <bra@fsn.hu> wrote:
Hi,
On 01/20/14 15:30, Laurens Van Houtven wrote:
The first answer by JP Calderone (exarkun) on the following stackoverflow question may be of some assistance.
http://stackoverflow.com/questions/10077745/twistedweb-on-multicore-multipro... What is the best way to use this capability from a .tac file?
There's nothing special about a .tac file with respect to spawning processes, it's just another way to instantiate a Service object. You just need to change the spawned process's arg list from "[executable, __file__, str(port.fileno())]" to something like "[executable, 'twistd', '-n', '-y', __file__, str(port.fileno())]". -glyph
On 03/12/15 18:42, Glyph Lefkowitz wrote:
On Mar 11, 2015, at 11:57 PM, Nagy, Attila <bra@fsn.hu> wrote:
Hi,
On 01/20/14 15:30, Laurens Van Houtven wrote:
The first answer by JP Calderone (exarkun) on the following stackoverflow question may be of some assistance.
http://stackoverflow.com/questions/10077745/twistedweb-on-multicore-multipro... What is the best way to use this capability from a .tac file? There's nothing special about a .tac file with respect to spawning processes, it's just another way to instantiate a Service object. You just need to change the spawned process's arg list from "[executable, __file__, str(port.fileno())]" to something like "[executable, 'twistd', '-n', '-y', __file__, str(port.fileno())]".
Sounds logical. Do you plan a nicer interface for this maybe? :)
On Mar 13, 2015, at 3:03 AM, Nagy, Attila <bra@fsn.hu> wrote:
Sounds logical. Do you plan a nicer interface for this maybe? :)
Plan? Not right now. Hope for? Yes, for many years :). I would very much like to have Twisted contain a straightforward process-pool interface, perhaps a deferToProcess. But attempting to guess when this will happen is a fool's errand. Perhaps you could help us do it, though! :) -glyph
On 03/13/15 11:28, Glyph wrote:
On Mar 13, 2015, at 3:03 AM, Nagy, Attila <bra@fsn.hu <mailto:bra@fsn.hu>> wrote:
Sounds logical. Do you plan a nicer interface for this maybe? :)
Plan? Not right now. Hope for? Yes, for many years :).
I would very much like to have Twisted contain a straightforward process-pool interface, perhaps a deferToProcess. But attempting to guess when this will happen is a fool's errand. Perhaps you could help us do it, though! :)
If I had the time. :) Anyways, big thanks for letting this done!
participants (5)
-
Glyph -
Glyph Lefkowitz
-
Laurens Van Houtven -
Nagy, Attila -
Sumanth Puram