[Twisted-Python] using spread/perspective broker
First off, the main reason I'm looking at this is more for perception than real need. My application is geared for medium/larger sized companies and those people want to know that the software is going to take advantage of their multiple cpu servers. In reality it's an IO bound application and cpu usage would just not be an issue from what I can tell. In any case I'm looking at ways to make it possible for it to run on multiple cpu's by being able to start up several processes and have them share the load. It looks like perspective broker is the right tool for the interprocess communication. My thought was that in a pool of processes, you would designate one of them to handle incoming connections and distribute them to other processes, which process the request, hand it back to the master, which sends the data back to the client. In addition, there should be some way to have the slave processes upgrade themselves to a master if the master process dies. This is all way more complicated than I would like though, and I'm hoping there is possibly a simpler solution. Any ideas?
On Tue, 2005-02-15 at 09:47 -0800, snacktime wrote:
This is all way more complicated than I would like though, and I'm hoping there is possibly a simpler solution. Any ideas?
Use one process, and any CPU intensive in a thread pool? If the CPU intensive bit involves C code it will use multiple CPUs.
Use one process, and any CPU intensive in a thread pool? If the CPU intensive bit involves C code it will use multiple CPUs.
There really isn't anything cpu intensive. The application is a simple line based protocol with ssl that accepts multiple requests per session to keep the ssl connection startup overhead to a minimum. It does some string formatting, an https POST to another server, a couple of database queries, and it's done. I was throwing around 100 transactions per second at it and it wasn't even using 1% of the cpu. After thinking about this I've decided not to worry so much about the perception some people might have. When you say C code will use multiple cpu's, do you mean that if a module is written in C, it no longer uses the GIL? I seem to remember something along these lines but I haven't looked into all the details of when the GIL comes into play and when it doesn't. Chris
On Feb 15, 2005, at 15:35, snacktime wrote:
Use one process, and any CPU intensive in a thread pool? If the CPU intensive bit involves C code it will use multiple CPUs.
There really isn't anything cpu intensive. The application is a simple line based protocol with ssl that accepts multiple requests per session to keep the ssl connection startup overhead to a minimum. It does some string formatting, an https POST to another server, a couple of database queries, and it's done. I was throwing around 100 transactions per second at it and it wasn't even using 1% of the cpu.
After thinking about this I've decided not to worry so much about the perception some people might have.
When you say C code will use multiple cpu's, do you mean that if a module is written in C, it no longer uses the GIL? I seem to remember something along these lines but I haven't looked into all the details of when the GIL comes into play and when it doesn't.
The GIL is always used, but you can explicitly release and acquire it from C code if you're not doing anything with PyObjects for a period of time (i.e. waiting for I/O, doing computation, etc.). -bob
participants (3)
-
Bob Ippolito -
Itamar Shtull-Trauring -
snacktime