On Mon, 18 Nov 2013 16:25:07 -0600 Tim Peters <tim.peters@gmail.com> wrote:
[Antoine]
Are very small pickles that size-sensitive? I have the impression that if 8 bytes vs. e.g. 15 bytes makes a difference for your application, you'd be better off with a hand-made format.
The difference between 8 and 15 is, e.g., nearly doubling the amount of network traffic (for apps that use pickles across processes or machines).
A general approach has no way to guess how it will be used. For example, `multiprocessing` uses pickles extensively for inter-process communication of Python data. Some users try broadcasting giant arrays across processes, while others broadcast oceans of tiny integers (like indices into giant arrays inherited via fork()).
Well, sending oceans of tiny integers will also incur many system calls and additional synchronization costs, since sending data on a multiprocessing Queue has to acquire a semaphore. So it generally sounds like a bad idea, IMHO. That said, I agree with:
Since pickle intends to be "the" Python serialization format, it really should try to be friendly for all plausible uses.
I simply don't think adding a fixed 8-byte overhead is actually annoying. It's less than the PyObject overhead in 64-bit mode... Regards Antoine.