performance of pickling & large lists :-(

Martin v. Loewis martin at v.loewis.de
Tue Aug 6 15:42:35 EDT 2002


Jeff Epler <jepler at unpythonic.net> writes:

> Python's "pickle" will use approximately one dict entry per object, of
> the form
>     { id(obj) : obj }
> this is used to make sure that a structure like 'y' in 
>     y = [[]]*2
> is pickled correctly, and that when reloaded, 'y[0] is y[1]'.
> 
> This accounts for most, if not all, of pickle's memory usage.

Unfortunately, I think this is incorrect: Pickle uses a dictionary of
the form

  { id(obj) : (last_position, obj) }

I think the majority of pickle's memory consumption comes from the
tuples.

Of course, the last_position field is need so that you know how to
pickle the backwards reference.

Regards,
Martin




More information about the Python-list mailing list