Pickling Robustly

[Martijn Faassen]

Glyph Lefkowitz <glyph at twistedmatrix.com> wrote:

[snip pickle is cool]
> Except...

> It appears that Pickle is not terribly robusta

[snip]
> Due to the extremely self-referential nature of the data structures
> I'm using, it is much easier to keep everything in one pickle (and
> that effectively happens anyway, even if I don't try to, thanks to the
> dependency resolution that pickle does). The problem is that pickle
> (as far as I can tell, anyway) does not allow one to register error
> handlers to keep track of when a problem occurrs.  If an unpicklable
> datatype is attempted to pickle, EVERYTHING previously pickled will
> suddenly vanish and no indication will be given about where the
> offending data was.

> If you've got a large data structure like I
> anticipate, this means writing and loading code on-the-fly to attempt
> a traversal of that structure to determine where the unpicklable type
> is.

The Digicool people must've run into similar problems with their Zope Object
Database, which is based on Pickle. Perhaps looking at the ZODB wouldn't
be a bad idea, if you haven't already, therefore:

http://www.python.org/workshops/2000-01/proceedings/papers/fulton/zodb3.html
http://www.zope.org/Members/michel/HowTos/ZODB-How-To
http://www.zope.org/Documentation/Developer/Models/ZODB

Though I'm not sure if they indeed have solved your problem, as far
as I understand it.

Regards,

Martijn