Multiprocessing.connection magic

[Chris Angelico]

On Fri, Jun 3, 2011 at 4:28 PM, Claudiu Popa <cpopa at bitdefender.com> wrote:
> Hello guys,
>      While  working  at a dispatcher using
>  multiprocessing.connection.Listener  module  I've stumbled upon some
>  sort    of  magic  trick  that  amazed  me. How is this possible and
>  what  does  multiprocessing  library doing in background for this to
>  work?

I'm not sure what magic trick you're referring to - is it that you can
use dissimilar versions of Python and send strange objects? I did find
that trying this in reverse (sending from 3.2, receiving with 2.7.1)
failed with "ValueError: unsupported pickle protocol: 3". That, plus
the docstring for send and recv, might suggest what's happening: the
object gets pickled.

Pickling in 2.7.1 (close enough to your 2.6 I presume):
http://docs.python.org/library/pickle.html
Pickling in 3.2:
http://docs.python.org/py3k/library/pickle.html

>From the 3.2 docs:
"The pickle serialization format is guaranteed to be backwards
compatible across Python releases."
"Protocol version 3 was added in Python 3.0. It has explicit support
for bytes and cannot be unpickled by Python 2.x pickle modules. This
is the current recommended protocol, use it whenever it is possible."
"The following types can be pickled:
...
functions defined at the top level of a module
built-in functions defined at the top level of a module
...
"

Presumably, the send() function pickles at the HIGHEST_PROTOCOL or
DEFAULT_PROTOCOL, and recv() unpickles whatever it gets. If you do the
pickling manually, you could choose to use version 2 explicitly, and
then the 2.6 other end could read it comfortably.

I don't know how effective the pickling of functions actually is.
Someone else will doubtless be able to fill that in.

Chris Angelico