[Python-Dev] Unpickling memory usage problem, and a proposed solution
Dan Gindikin
dgindikin at gmail.com
Fri Apr 23 23:50:50 CEST 2010
Antoine Pitrou <solipsis <at> pitrou.net> writes:
> Does cPickle bytecode have some kind of NOP instruction?
> You could keep track of which PUTs weren't necessary and zero them out at the
> end. It would be much cheaper than writing a whole other "optimized" stream.
For a large file, I'm not sure it is much faster to edit it in place
than to rewrite it, and also since it's a large file, you are going
to probably have it compressed, in which case you are out of luck anyway.
> (of course it only works on a seekable stream )
Indeed... since I was dealing with zipped files, I had to pass in
a "seek0" func, like so:
file_in_seek0_func = lambda x: os.popen('zcat ' + file_in)
More information about the Python-Dev
mailing list