[Python-ideas] Copy-on-write when forking a python process

[jac]

> But maybe it's not premature. Do you have measurements that show how
> much extra swap space is taken up byCOWcopies caused by changing
> reference counts in your application?

In my case, memory almost the entire size of the dictionary is being
copied into the child process.  But for any specific case there will
be several factors involved:
The size of a page on the os
How many elements of the dictionary are accessed
The size of the objects in the dictionary (keys and values)
The distribution of the objects in memory. (For small objs you may
have more then one on a page, etc.)
etc.

But I think that it is possible in some cases that *more* memory then
the entire dictionary will be copied into the child process' memory.
I think that this would happen if each key value pair of the
dictionary were to be iterated over, and if the page size of the os
was larger then the size of an object, and the objects were arranged
in memory such that no two objects were contiguous.

--jac

On Apr 13, 3:34 am, Mike Meyer <m... at mired.org> wrote:
> On Tue, 12 Apr 2011 14:42:43 -0700 (PDT)
>
> jac <john.theman.con... at gmail.com> wrote:
> > Hi all,
> > Sorry for cross posting, but I think that this group may actually be
> > more appropriate for this discussion.  Previous thread is at:
> >http://groups.google.com/group/comp.lang.python/browse_thread/thread/...
>
> > I am wondering if anything can be done about theCOW(copy-on-write)
> > problem when forking a python process.  I have found several
> > discussions of this problem, but I have seen no proposed solutions or
> > workarounds.  My understanding of the problem is that an object's
> > reference count is stored in the "ob_refcnt" field of the PyObject
> > structure itself.  When a process forks, its memory is initially not
> > copied. However, if any references to an object are made or destroyed
> > in the child process, the page in which the objects "ob_refcnt" field
> > is located in will be copied.
>
> This smells like premature optimization to me. You're worried about
> the kernel copying a few extra pages of user data when you're dealing
> with a dictionary that's gigabytes in size. Sounds like any possibly
> memory savings here would be much smaller than those that could come
> from improving the data encoding.
>
> But maybe it's not premature. Do you have measurements that show how
> much extra swap space is taken up byCOWcopies caused by changing
> reference counts in your application?
>
>       <mike
> --
> Mike Meyer <m... at mired.org>              http://www.mired.org/consulting.html
> Independent Software developer/SCM consultant, email for more information.
>
> O< ascii ribbon campaign - stop html mail -www.asciiribbon.org
> _______________________________________________
> Python-ideas mailing list
> Python-id... at python.orghttp://mail.python.org/mailman/listinfo/python-ideas