[Python-ideas] solving multi-core Python

[Devin Jeanpierre]

On Tue, Jun 23, 2015 at 7:55 AM, Sturla Molden <sturla.molden at gmail.com> wrote:
> On 23/06/15 15:14, Jonas Wielicki wrote:
>
>> To be fair, you will nevertheless get a slowdown when copy-on-write
>> kicks in while first using whatever was cloned from the parent. This is
>> nothing which blocks execution, but slows down execution.
>
>
> Yes, particularly because of reference counts. Unfortunately Python stores
> refcounts within the PyObject struct. And when a refcount is updated a copy
> of the entire 4 KB page is triggered. There would be fare less of this if
> refcounts was kept in dedicated pages.

A coworker of mine wrote a patch to Python that allows you to freeze
refcounts for all existing objects before forking, if the correct
compile options are set. This adds overhead to incref/decref, but
dramatically changes the python+fork memory usage story. (I haven't
personally played with it much, but it sounds decent.) If there's any
interest I can try to upstream this change, guarded behind a compiler
flag.

We've also tried moving refcounts to their own pages, like you and
Nick suggest, but it breaks a *lot* of third-party code. I can try to
upstream it. If it's guarded by a compiler flag it is probably still
useful, just any users would have to grep through their dependencies
to make sure nothing directly accesses the refcount. (The stdlib can
be made to work.)  It sounds like it would also be useful for the main
project in the topic of this thread, so I imagine there's more
momentum behind it.

-- Devin