[Python-ideas] solving multi-core Python

[Sturla Molden]

On 22/06/15 01:39, Antoine Pitrou wrote:

> Or perhaps the fact that those superiors APIs are a PITA.

Not all of them, no.

HeapAlloc is a good example. Very easy to use, and the "one heap per 
thread" design often gives excellent performance compared to a single 
global heap. But on Linux we only have malloc et al., allocating from 
the global heap. How many Linux programmers have even considered using 
multiple heaps in combination with multi-threading? I can assure you it 
is not common.

A good idea is to look at the Python C API. We have PyMem_Malloc, but 
nothing that compares to Windows' HeapAlloc.

Not only does HeapAlloc remove the contention for the global heap, it 
can also serialize. Instead of serializing an object by traversing all 
references in the object tree, we just serialize the heap from which it 
was allocated.

And as for garbage collection, why not deallocate the whole heap in one 
blow? Is the any reason to pair each malloc with free if one could just 
zap the whole heap? That is what HeapDestroy does. On Linux we would 
typically homebrew a memory pool to achieve the same thing. But a memory 
pool needs to traverse a chain of pointers and call free() multiple 
times, each time with contention for the spinlock protecting the global 
heap. And when allocating from a memory pool we also have contention for 
the global heap. It cannot in any way compare to the performance of the 
Win API HeapCreate/HeapDestroy and HeapAlloc/HeapFree.


Sturla