On Tue, Jun 23, 2015 at 7:55 AM, Sturla Molden <sturla.molden@gmail.com> wrote:
On 23/06/15 15:14, Jonas Wielicki wrote:
To be fair, you will nevertheless get a slowdown when copy-on-write kicks in while first using whatever was cloned from the parent. This is nothing which blocks execution, but slows down execution.
Yes, particularly because of reference counts. Unfortunately Python stores refcounts within the PyObject struct. And when a refcount is updated a copy of the entire 4 KB page is triggered. There would be fare less of this if refcounts was kept in dedicated pages.
A coworker of mine wrote a patch to Python that allows you to freeze refcounts for all existing objects before forking, if the correct compile options are set. This adds overhead to incref/decref, but dramatically changes the python+fork memory usage story. (I haven't personally played with it much, but it sounds decent.) If there's any interest I can try to upstream this change, guarded behind a compiler flag. We've also tried moving refcounts to their own pages, like you and Nick suggest, but it breaks a *lot* of third-party code. I can try to upstream it. If it's guarded by a compiler flag it is probably still useful, just any users would have to grep through their dependencies to make sure nothing directly accesses the refcount. (The stdlib can be made to work.) It sounds like it would also be useful for the main project in the topic of this thread, so I imagine there's more momentum behind it. -- Devin