[Python-Dev] Unpickling memory usage problem, and a proposed solution

[Dan Gindikin]

Antoine Pitrou <solipsis <at> pitrou.net> writes:
> Does cPickle bytecode have some kind of NOP instruction?
> You could keep track of which PUTs weren't necessary and zero them out at the
> end. It would be much cheaper than writing a whole other "optimized" stream.

For a large file, I'm not sure it is much faster to edit it in place
than to rewrite it, and also since it's a large file, you are going
to probably have it compressed, in which case you are out of luck anyway.

> (of course it only works on a seekable stream )

Indeed... since I was dealing with zipped files, I had to pass in
a "seek0" func, like so:

   file_in_seek0_func = lambda x: os.popen('zcat ' + file_in)