Pickling buffer objects.
Before submitting a patch to pickle.py and cPickle.c, I'd be interested in knowing how likely to be accepted a patch that allows Python to pickle the buffer object. The problem being solved is that Numeric currently has to copy all of its data into a string before writing it out to a pickle. Yes, I know there are ways to write directly to a file. But, it is desireable to have Numeric arrays interact seamlessly with other pickleable types without a separate stream. This is especially utilized for network transport. The patch would simply write the opcode for a Python string to the stream and then write the character-interpreted data (without making an intermediate copy) of the void * pointer of the buffer object. Yes, I know all of the old arguments about the buffer object and that it should be replaced with something better. I've read all the old posts and am quite familiar with the issues about it. But, this can be considered a separate issue. Since the buffer object exists, it ought to be pickleable, and it would make a lot of applications a lot faster. I'm proposing to pickle the buffer object so that it unpickles as a string. Arguably, there should be a separate mutable-byte object opcode so that buffer objects unpickle as mutable-byte buffer objects. If that is more desireable, I'd even offer a patch to do that (though such pickles wouldn't unpickle under earlier versions of Python). I suspect that the buffer object would need to be reworked into something more along the lines of the previously-proposed bytes object before a separate bytecode for pickleable mutable-bytes is accepted, however. -Travis Oliphant
Travis Oliphant wrote:
I'm proposing to pickle the buffer object so that it unpickles as a string.
Wouldn't this mean you're only solving half the problem? Unpickling a Numeric array this way would still use an intermediate string. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg.ewing@canterbury.ac.nz +--------------------------------------+
Greg Ewing wrote:
Travis Oliphant wrote:
I'm proposing to pickle the buffer object so that it unpickles as a string.
Wouldn't this mean you're only solving half the problem? Unpickling a Numeric array this way would still use an intermediate string.
Well, actually, unpickling in the new numeric uses the intermediate string as the memory (yes, I know it's not supposed to be "mutable", but without a mutable bytes object what else are you supposed to do?). Thus, ideally we would have a mutable-bytes object with a separate pickle opcode. Without this, then we overuse the string object. But, since the string is only created by the pickle (and nobody else uses it, then what's the real harm). So, in reality the previously-mentioned patch together with modificiations to Numeric's unpickling code actually solves the whole problem. -Travis
participants (3)
-
"Martin v. Löwis"
-
Greg Ewing
-
Travis Oliphant