On Fri, Aug 20, 2021 at 5:22 PM <c.buhtz@posteo.jp> wrote:
I simply tried to understand how processes transfering data between each other. I know they pickle. But how exactly? Which pickle protocol they use by default? Do they decide the protocol depending on the type/kind/structure of data? Do they compress the pickled data? e.g. I read a PEP about pickle version 5 which is relevant for large data like pandas.DataFrames.
pickle.DEFAULT_PROTOCOL would be my first guess :) When you're curious about this sort of thing, I strongly recommend browsing the CPython source code. Sometimes, you'll end up with a follow-up question "is this a language guarantee?", but at very least, you'll know how the most-used Python implementation does things. Don't be put off by the "C" in CPython; a lot of the standard library is implemented in Python, including the entire multiprocessing module: https://github.com/python/cpython/tree/main/Lib/multiprocessing A quick search for the word "pickle" shows this as a promising start: https://github.com/python/cpython/blob/main/Lib/multiprocessing/reduction.py ChrisA