Overcoming python performance penalty for multicore CPU

[Antoine Pitrou]

Le Tue, 02 Feb 2010 15:02:49 -0800, John Nagle a écrit :
> I know there's a performance penalty for running Python on a multicore
> CPU, but how bad is it?  I've read the key paper
> ("www.dabeaz.com/python/GIL.pdf"), of course.  It would be adequate if
> the GIL just limited Python to running on one CPU at a time, but it's
> worse than that; there's excessive overhead due to a lame locking
> implementation.  Running CPU-bound multithreaded code on a dual-core CPU
> runs HALF AS FAST as on a single-core CPU, according to Beasley.

That's on certain types of workloads, and perhaps on certain OSes, so you 
should try benching your own workload to see whether it applies.

Two closing remarks:
- this should (hopefully) be fixed in 3.2, as exarkun noticed
- instead of spawning one thread per Web page, you could use Twisted or 
another event loop mechanism in order to process pages serially, in the 
order of arrival

Regards

Antoine.