Hello,

I have been working on the concurent.futures module lately and I think this optimization should be avoided in the context of python Pools.

This is an interesting idea, however its implementation will bring many complicated issues as it breaks the basic paradigm of a Pool: the tasks are independent and you don't know which worker is going to run which task.

The function is serialized with each task because of this paradigm. This ensure that any worker picking the task will be able to perform it independently from the tasks it has run before, given that it as been initialized correctly at the beginning. This makes it simple to run each task.

As the Pool comes with no scheduler, with your idea, you would need a synchronization step to send the function to all workers before you can launch your task. But if there is already one worker performing a long running task, does the Pool wait for it to be done before it sends the function? If the Pool doesn't wait, how does it ensure that this worker will be able to get the definition of the function before running it?

Also, the multiprocessing.Pool has some features where a worker can shut itself down after a given number of tasks or a timeout. How does it ensure that the new worker will have the definition of the function?

It is unsafe to try such a feature (sending only once an object) anywhere else than in the initializer which is guaranteed to be run once per worker.

On the other hand, you mentioned an interesting point being that making globals available in the workers could be made simpler. A possible solution would be to add a "globals" argument in the Pool which would instanciate global variables in the workers. I have no specific idea but on the implementation of such features but it would be safer as it would be an initialization feature.