[Python-Dev] Pickling w/ low overhead
Tue, 3 Aug 1999 13:58:23 -0700 (Pacific Daylight Time)
On Tue, 3 Aug 1999, Skip Montanaro wrote:
> Using __getstate__/__setstate__, could you create a compressed
> representation using zlib or some other scheme? I don't know how well
> numeric data compresses in general, but that might help. Also, I trust you
> use cPickle when it's available, yes?
I *really* hate to admit it, but I've found the source of the most massive
problem in the pickling process that I was using. I didn't use binary
mode, which meant that the huge strings were written & read
I think I'll put a big fat note in the NumPy doc to that effect.
(note that luckily this just affected my usage, not all NumPy users).
<embarassed sheepish grin>