[Python-Dev] Accepting PEP 3154 for 3.4?

[Antoine Pitrou]

On Mon, 18 Nov 2013 16:25:07 -0600
Tim Peters <tim.peters at gmail.com> wrote:
> 
> [Antoine]
> > Are very small pickles that size-sensitive? I have the impression that
> > if 8 bytes vs. e.g. 15 bytes makes a difference for your application,
> > you'd be better off with a hand-made format.
> 
> The difference between 8 and 15 is, e.g., nearly doubling the amount
> of network traffic (for apps that use pickles across processes or
> machines).
> 
> A general approach has no way to guess how it will be used.  For
> example, `multiprocessing` uses pickles extensively for inter-process
> communication of Python data.  Some users try broadcasting giant
> arrays across processes, while others broadcast oceans of tiny
> integers (like indices into giant arrays inherited via fork()).

Well, sending oceans of tiny integers will also incur many system calls
and additional synchronization costs, since sending data on a
multiprocessing Queue has to acquire a semaphore. So it generally
sounds like a bad idea, IMHO.

That said, I agree with:
> Since pickle intends to be "the" Python serialization format, it
> really should try to be friendly for all plausible uses.

I simply don't think adding a fixed 8-byte overhead is actually
annoying. It's less than the PyObject overhead in 64-bit mode...

Regards

Antoine.