[Python-ideas] Copy-on-write when forking a python process

[Nick Coghlan]

On Wed, Apr 13, 2011 at 7:42 AM, jac <john.theman.connor at gmail.com> wrote:
> Has anyone else looked into the COW problem?  Are there workarounds
> and/or other plans to fix it?  Does the solution I am proposing sound
> reasonable, or does it seem like overkill?  Does anyone see any
> (technical) problems with it?

There's a clear workaround for the COW problem these days: use PyPy
instead of CPython :)

Currently that workaround comes at a potentially high cost in
compatibility with 3rd party C extensions, but that situation will
naturally improve over time. Given that a lot of those compatibility
problems arise *because* PyPy doesn't use refcounting natively, it's
highly unlikely that there will be any significant tinkering with
CPython's own approach.

As far as technical problems go, opting out of memory management is a
beautiful way to shoot yourself in the foot with memory leaks. All it
takes is one optout() without a corresponding optin() and an arbitrary
amount of memory may fail to be released. For example, in your own
post, any exception in Fork_and_block_while_doing_stuff() means
anything referenced directly or indirectly from mylist will be left
hanging around in memory until the process terminates. That's a *far*
worse problem than being unable to readily share memory between
processes.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia