Am Mittwoch, den 06.01.2010, 22:02 -0500 schrieb Glyph Lefkowitz:
3) So, suppose I have one twisted reactor based process running,, I can use defferToThread as one of the way to kind of using multiple processing?
No, not really. Threads in python are bound by the global interpreter lock and are therefore not that useful for using multiple processors.
While this is generally true, in specific cases you can use it for speeding up by using multiple cores. One example would be a program that does image manipulation via PIL. PIL does release the GIL before it starts to munch on the image, hence you can achieve more than 100% user time :) One script that I use to scale images does manage, in ideal settings over 300% user time; the trick being to use a couple of threads more than I've got cpu cores, this way a couple of threads can inform the Linux kernel what files the process is interested in. This way, the CPU has usually enough work to do the work, and as most work is spent in C code in PIL, the GIL does not limit CPU utilization. But generally speaking as Glyph has mentioned, Python usually can use only one CPU core per process. The reason for that you need to hold a single global lock to access the virtual machine, but also the runtime in any way. So some C function, can do something like that: * extract input arguments from PyObjects. * release GIL * munch munch * reacquire GIL * create PyObjects for return state. notice that only code that in no way touches the Python runtime can run without the GIL being hold. Andreas
_______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web