performance of pickling & large lists :-(
Martin v. Loewis
martin at v.loewis.de
Tue Aug 6 15:42:35 EDT 2002
Jeff Epler <jepler at unpythonic.net> writes:
> Python's "pickle" will use approximately one dict entry per object, of
> the form
> { id(obj) : obj }
> this is used to make sure that a structure like 'y' in
> y = [[]]*2
> is pickled correctly, and that when reloaded, 'y[0] is y[1]'.
>
> This accounts for most, if not all, of pickle's memory usage.
Unfortunately, I think this is incorrect: Pickle uses a dictionary of
the form
{ id(obj) : (last_position, obj) }
I think the majority of pickle's memory consumption comes from the
tuples.
Of course, the last_position field is need so that you know how to
pickle the backwards reference.
Regards,
Martin
More information about the Python-list
mailing list