No limit! (was: pickle complexity limit?)

[Christian Tismer]

Martin v. Löwis wrote:

> "Mark Hahn" <mark at hahnca.com> writes:
> 
> 
>>I don't understand how this could happen with pickle.  Isn't it supposed to
>>stop when it runs into an object it has already pickled?  
> 
> Yes, and it does.
> 
>>Couldn't it have been coded in a stackless way?
> 
> Yes, but it doesn't. Please look at save_inst implementation; it ends
> with
> 
>         save(stuff)
>         write(BUILD)
> 
> This is where the recursion happens. You run into the limit only if
> you can find a path through your object graph that does not visit the
> same object twice.
> 
>>What do I do now?   I don't want to recompile with a deeper stack as this
>>will obviously only help in limited cases, not solve the real problem.

There are limitations, and there are ways to cope with them. See below.

> Read the pickle code, understand why it is recursive, and implement an
> alternative that isn't recursive, yet preserves the original semantics
> in terms of pickling order and sequence in which methods are called on
> objects. When you are done, please contribute your changes back, to
> sf.net/projects/python.

The reason why pickling is recursive was of course a thing
that I never really understood. For me, this is a bit like
asking for problems.
But well, the same was true with recursive destructors of
objects, until I came up with the Py_Trashcan structure.
It has been changed very much, but the idea is still alive:
try to limit recursions as much as possible.

Back to your problem: You have a chance to avoid recursions.
I had the same problem with Stackless. I support pickling of
frame chains to any extent, together with tracebacks. The
recursion problems would be there, if I didn't take some action.
ANd for Stackless, these problems are in fact harder, since
it doesn't have any restriction in recursion depth by nature.

Let me give you an example from my recent efforts:

For Stackless, I implemented pickling of frames and tracebacks,
besides many others. With the standard approach of pickling
them as they come, I would have got into recursion problems.
Although I got different advice of Armin Rigo ("do it right,
trash the pickler and build a stackless engine for it"), I didn't,
since I'm not trying to convince the stack-bound heads any longer.
I will do that for PyPy, when the time has come.

Keeping the current recursive pickling engine, I did this for
pickling of Tasklets, which are just small header objects
for frame chains:
Frames are pickled alone. They don't care about their f_back
field, therefore not introducing any recursive calls.
Tasklets are frame chains. They know which frames they
contain. And instead of causing recursions, tasklets simply
build a list of all the frames they need to be pickled,
and exactly that will happen, in an iterative manner.

Analogously it works with tracebacks, which also can show
very serious nesting. The problem with tracebacks is a bit
harder than with tasklets, since they don't have the
tasklet serving like a header. Therefore, I added a flag to
tracebacks, which marks the leading traceback object. That is,
this leading one will pickle all the other ones into a list,
again avoiding deep recursions. The trick was to modify
PyTraceBack_Here to tag the leading traceback object. Under the
(reasonable) assumption that this leading traceback object
will show up in your pickle, things work very smoothly.

If you want to know about how to pickle certain stuff, download
the Stackless CVS and have a look. There is support for pickling
tasklets, frames, code objects, iterators, tracebacks, and some
more to come. The most straight forward thing for you to look
into would probably be tracebackobject.c .

CVSROOT=:pserver:anonymous at centera.de:/home/cvs
export CVSROOT
cvs co stackless

cheers - chris

-- 
Christian Tismer             :^)   <mailto:tismer at tismer.com>
Mission Impossible 5oftware  :     Have a break! Take a ride on Python's
Johannes-Niemeyer-Weg 9a     :    *Starship* http://starship.python.net/
14109 Berlin                 :     PGP key -> http://wwwkeys.pgp.net/
work +49 30 89 09 53 34  home +49 30 802 86 56  mobile +49 173 24 18 776
PGP 0x57F3BF04       9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
      whom do you want to sponsor today?   http://www.stackless.com/