Overcoming python performance penalty for multicore CPU
ryan at rfk.id.au
Sun Feb 21 22:45:50 CET 2010
On Sun, 2010-02-21 at 22:22 +0100, Martin v. Loewis wrote:
> John Nagle wrote:
> > I know there's a performance penalty for running Python on a
> > multicore CPU, but how bad is it? I've read the key paper
> > ("www.dabeaz.com/python/GIL.pdf"), of course. It would be adequate
> > if the GIL just limited Python to running on one CPU at a time,
> > but it's worse than that; there's excessive overhead due to
> > a lame locking implementation. Running CPU-bound multithreaded
> > code on a dual-core CPU runs HALF AS FAST as on a single-core
> > CPU, according to Beasley.
> I couldn't reproduce these results on Linux. Not sure what "HALF AS
> FAST" is; I suppose it means "it runs TWICE AS LONG" - this is what I
> couldn't reproduce.
> If I run Beazley's program on Linux 2.6.26, on a 4 processor Xeon (3GHz)
> machine, I get 30s for the sequential execution, 40s for the
> multi-threaded case, and 32s for the multi-threaded case when pinning
> the Python process to a single CPU (using taskset(1)).
> So it's 6% overhead for threading, and 25% penalty for multicore CPUs -
> far from the 100% you seem to expect.
It's far from scientific, but I've seen behaviour that's close to a 100%
performance penalty on a dual-core linux system:
Short story: a particular test suite of mine used to run in around 25
seconds, but a bit of ctypes magic to set thread affinity dropped the
running time to under 13 seconds.
http://www.rfk.id.au | This message is digitally signed. Please visit
ryan at rfk.id.au | http://www.rfk.id.au/ramblings/gpg/ for details
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 197 bytes
Desc: This is a digitally signed message part
More information about the Python-list