Benjamin Root <ben.v.root@gmail.com> wrote:
Oftentimes, if one needs to share numpy arrays for multiprocessing, I would imagine that it is because the array is huge, right?
That is a case for shared memory, but what. i was taking about is more common than this. In order for processes to cooperate, they must communicate. So we need a way to pass around NumPy arrays quickly. Sometimes we want to use shared memory because of the size of the data, but more often it is just used as a form of inexpensive IPC.
So, the pickling approach would copy that array for each process, which defeats the purpose, right?
I am not sure what you mean. When I made shared memory arrays I used named segments, and made sure only the name of the segments were pickled, not the contents of the buffers. Sturla