[C++-sig] [pygccxml] caching fails due to pickling recursion limit
kir at lapshin.net
Thu Jun 28 17:57:58 CEST 2007
Hi Roman, everybody,
First of all my apologies if it is a wrong list for pygccxml discussion.
We are using py++/pygccxml on moderately sized project, and recently
py++ started to fail silently on us. It runs gccxml and then silently quits.
It turned out the offending piece of code was pickling cache. Something
fails in cPickle and terminates the whole process. Most likely it is a
stack overflow, because if cPickle replaced with pickle, then we get
recursion limit runtime error. One can raise recursion limit and pickle
will save cache just fine, but really slowly (not surprising at all).
At the moment we've worked around this problem by disabling cache,
however it would be nice to fix it properly.
After reading a bit about pickling, it looks like (c)pickle don't work
very well with recursive structures, which declarations obviously are.
I just want to list some ideas I have on how to tackle this problem, and
I would love to get some comments on their feasibility, or better yet
1. Simple one, just set all parents to None before pickling and rebuild
parents upon loading. Should be relatively simple, I'll try it first
thing. Hopefully that would kill fair amount of recursion, but not all
of it (e.g. class A refers to class B and B to A).
2. Add unique id to each declaration, have a global store that saves
declarations by id, and define for each declaration __getstate__,
__setstate__ that would serialize ids, rather then declaration
instances. Then we just pickle the global store. In reality somewhat
more complex approach needed to tackle recursive references.
Reversible flatting out:
Some details: we are running:
Python 2.4.3 on Windows XP
gccxml -- somewhat old CVS snapshot
More information about the Cplusplus-sig