[Python-Dev] Pickling buffer objects.

[Travis Oliphant]

Before submitting a patch to pickle.py and cPickle.c,  I'd be interested 
in knowing how likely to be accepted a patch that allows Python to 
pickle the buffer object.

The problem being solved is that Numeric currently has to copy all of 
its data into a string before writing it out to a pickle.  Yes, I know 
there are ways to write directly to a file.  But,  it is desireable to 
have Numeric arrays interact seamlessly with other pickleable types 
without a separate stream.   This is especially utilized for network 
transport. 

The patch would simply write the opcode for a Python string to the 
stream and then write the character-interpreted data (without making an 
intermediate copy) of the void * pointer of the buffer object. 

Yes, I know all of the old arguments about the buffer object and that it 
should be replaced with something better.    I've read all the old posts 
and am quite familiar with the issues about it.

But, this can be considered a separate issue.  Since the buffer object 
exists, it ought to be pickleable, and it would make a lot of 
applications a lot faster.  

I'm proposing to pickle the buffer object so that it unpickles as a 
string.  Arguably, there should be a separate mutable-byte object opcode 
so that buffer objects unpickle as mutable-byte buffer objects.   If 
that is more desireable, I'd even offer a patch to do that (though such 
pickles wouldn't unpickle under earlier versions of Python).   I suspect 
that the buffer object would need to be reworked into something more 
along the lines of the previously-proposed bytes object before a 
separate bytecode for pickleable mutable-bytes is accepted, however.

-Travis Oliphant