I have to say thank you very much for all your experience and hints. It helps me a lot just talking with real professionals! ;) On 2021-08-19 23:32 "Stephen J. Turnbull" <turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
The remaining question is "how many chunks?" If that's relevant, ISTM a few simple experiments will show where the sweet spot is. Try a queue of 64 chunks, then 128 chunks, and refine guesses from there.
That brings me to another side-question which brings me to an unanswered bug-report https://bugs.python.org/issue44901 This was reported as a bug by me because I see this as a problem with documentation. I simply tried to understand how processes transfering data between each other. I know they pickle. But how exactly? Which pickle protocol they use by default? Do they decide the protocol depending on the type/kind/structure of data? Do they compress the pickled data? e.g. I read a PEP about pickle version 5 which is relevant for large data like pandas.DataFrames. Knowing more about this would help me to understand the needed ressources for creating a process and transfering data to and from it. The RAM itself does not take into account. We assume here I have enough RAM.