[Python-ideas] solving multi-core Python
sturla.molden at gmail.com
Mon Jun 22 02:41:49 CEST 2015
On 22/06/15 01:39, Antoine Pitrou wrote:
> Or perhaps the fact that those superiors APIs are a PITA.
Not all of them, no.
HeapAlloc is a good example. Very easy to use, and the "one heap per
thread" design often gives excellent performance compared to a single
global heap. But on Linux we only have malloc et al., allocating from
the global heap. How many Linux programmers have even considered using
multiple heaps in combination with multi-threading? I can assure you it
is not common.
A good idea is to look at the Python C API. We have PyMem_Malloc, but
nothing that compares to Windows' HeapAlloc.
Not only does HeapAlloc remove the contention for the global heap, it
can also serialize. Instead of serializing an object by traversing all
references in the object tree, we just serialize the heap from which it
And as for garbage collection, why not deallocate the whole heap in one
blow? Is the any reason to pair each malloc with free if one could just
zap the whole heap? That is what HeapDestroy does. On Linux we would
typically homebrew a memory pool to achieve the same thing. But a memory
pool needs to traverse a chain of pointers and call free() multiple
times, each time with contention for the spinlock protecting the global
heap. And when allocating from a memory pool we also have contention for
the global heap. It cannot in any way compare to the performance of the
Win API HeapCreate/HeapDestroy and HeapAlloc/HeapFree.
More information about the Python-ideas