Re: [Twisted-web] How does Twisted Web Multiprocess work ?

Hi Carl The only motivation to use multiple processes is to be able to parallelize blocking calls (since event driven handling is not useful) In my case forward proxy is not a bottleneck, however when my code spawns multiple processes as per http://stackoverflow.com/questions/10077745/twistedweb-on-multicore-multipro... and I run multiple concurrent clients, I see that the server's processes are not efficiently utilized. Specifically, even if there are outstanding requests, one of the processes accepts requests lot more while other processes are idling. I was able to confirm this by printing the Process IDs of the processes which are handling request This sort of imbalance could lead to underutilization of CPU - also discussed here https://lwn.net/Articles/542629/ Hence I was curious to know how Twisted multiprocessing works. It seems SO_REUSEPORT is not used within Twisted and as per Gavin's reply can be used "outside" of Twisted. On Wed, Jun 10, 2015 at 5:42 PM, Carl Waldbieser wrote:

There is a twisted developer guide for communicating with child processes: http://twistedmatrix.com/documents/current/core/howto/process.html

I'm not sure I understand why having multiple processes listen on the same socket is desirable in your case. From reading the articles you linked to, it seems like it is only useful in the case where a forward proxy becomes a bottleneck. Is that the case wirh your application?

Thanks, Carl Waldbieser On Jun 9, 2015 10:03 PM, "Sagar Dixit" wrote:

...

Hi,

I am exploring Twisted Web for my RESTful application. My application is stateless and involves storing and retrieving objects based on Object-ID. This application will run on beefy (multicore, lots of memory) machine. However, not all APIs that the application issues to underlying storage are async and hence I cannot fully utilize Deferreds Which means, there will some blocking calls and hence my primary interest is to use Twisted Web in multiprocessing mode

I came across http://stackoverflow.com/questions/10077745/twistedweb-on-multicore-multipro...

However, I am not sure if it is the "correct" way of doing things.

Hence I had some questions around it:

1. Is there an interface (similar to defertoThread) which allows me to execute a blocking call in a separate process ?

2. Does reactor synchronize access of all processes to the shared listen socket ?

3. Is there a sample code I can refer to where the application is spawning subprocesses to handle HTTP requests ?

Thanks in advance!

_______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web

_______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web

-- ssdixit