[Python-Dev] Unpickling memory usage problem, and a proposed solution

[Alexandre Vassalotti]

On Fri, Apr 23, 2010 at 3:07 PM, Collin Winter <collinwinter at google.com> wrote:
> I should add that, adding the necessary bookkeeping to remove only
> unused PUTs (instead of the current all-or-nothing scheme) should not
> be hard. I'd watch out for a further performance/memory hit; the
> pickling benchmarks in the benchmark suite should help assess this.

I was thinking about this too. A simple boolean table could be fast,
while keeping the space requirement down. This scheme would be nice to
caches as well.

> The current optimization penalizes pickling to speed up unpickling,
> which made sense when optimizing pickles that would go into memcache
> and be read out 13-15x more often than they were written.

This is my current impression of how pickle is most often used. Are
you aware of a use case of pickle where you do more writes than reads?
I can't think of any.

-- Alexandre